Skip to content

feat(telemetry): the numbers a spike needs are exported, alerted on, and reach the public health check (Scaling H5) - #2584

Merged
genekogan merged 2 commits into
mainfrom
agent/2541-alerts-reach
Nov 16, 2026
Merged

genekogan merged 2 commits into
mainfrom
agent/2541-alerts-reach

Conversation

@genekogan

Copy link
Copy Markdown
Member

Problem

During a spike nobody was told anything. Viewers per world, the launcher's own event-loop delay, worlds created and free disk were not exported. No unit ran the collector with alerts, and an alert that fired stayed a local line in alerts.jsonl. Closes #2541.

Acceptance

  • A1 Tests for each series and each rule (fire and resolve) — evidence: node --test tools/world-launcher/telemetry-spike.test.mjs -> 7 pass, 0 fail. Covered: the four series and their absence when unreported. launcher-loop-stall fires within 60 s and resolves below 0.15 s. worlds-created fires on 11 creations in an hour across a launcher restart, does not fire on 10, and resolves when the hour has passed. disk-free fires and resolves. viewers-high fires and resolves in alerts.jsonl and never pages.
  • A2 Test: a firing rule makes /__sim/healthz report hostOk: false with the code, and resolving clears it — evidence: node --test tools/world-launcher/telemetry-spike.test.mjs "A2: a firing rule makes /__sim/healthz report hostOk false with its code; resolving clears it at once; the launcher exports the series" -> pass. It runs a real launcher: the collector's own evaluation fires disk-free, healthz says hostOk: false, hostCodes: ["alert:disk-free"], and after the resolve hostOk is no longer false and hostCodes is gone. A stale timer copy does not hold the code, and a dead collector shows as alert:stale.
  • A3 docs/operations/telemetry.md documents the series, rules, the unit and how to enable it — evidence: docs/operations/telemetry.md sections "The series a spike needs", "Rules" (four new rows and which rules page), "Reaching a person" and "The unit" (enable, check and stop commands)

Proof

Tier: T1 (node tools/proof/tier.mjs --base origin/main -> T1)

  • proof.json: not required at T1
  • clips: none (no visible change)
  • images: none

Evaluation cost with the new rules (the issue asked for it again). The run used 400 registered worlds (20 running) and one synthetic hour: 240 scrapes, alerts evaluated after each. Base (the rules on main) and candidate ran back to back, two repeats each, on a shared 16-core workstation with a 1-min load of 5.6–7.6. These are in-process timings of one function, not a load run:

arm median p95 max
base, repeat 1 / 2 39.3 / 38.7 ms 69.0 / 68.3 ms 80.4 / 72.6 ms
candidate, repeat 1 / 2 40.9 / 40.7 ms 70.5 / 70.7 ms 79.3 / 75.9 ms

That is about 2 ms more per scrape, still about 0.3 % of one core at the 15 s cadence.

Self-audit

  • Checks run:
    • node --test tools/world-launcher/telemetry-spike.test.mjs tools/world-launcher/telemetry-alerts.test.mjs tools/world-launcher/telemetry.test.mjs tools/world-launcher/metrics.test.mjs tools/world-launcher/host-health.test.mjs tools/world-launcher/health-dash.test.mjs tools/world-launcher/health-dash-backup.test.mjs tools/world-launcher/health-dash-auth.test.mjs tools/world-launcher/telemetry-cost.test.mjs tools/world-launcher/telemetry-s2.test.mjs tools/world-launcher/telemetry-units.test.mjs tools/world-launcher/ceilings.test.mjs tools/world-launcher/guard.test.mjs tools/world-launcher/account-creation.test.mjs tools/world-launcher/heap.test.mjs tools/world-launcher/ship-root.test.mjs -> 126 pass, 0 fail, 0 skipped
    • node --test tools/world-launcher/keeper.test.mjs tools/world-launcher/launcher.test.mjs tools/world-launcher/shed.test.mjs -> pass (after the fix under Failed attempts)
    • bash -n tools/world-launcher/deploy/install.sh -> ok
    • node .github/scripts/guards.mjs scrub --base origin/main -> 0 problems
    • Unverified: the unit on a real host. Enabling it is the maintainer's step, and nothing here touched a host.
  • Failed attempts:
    • The first version put the alerts-file reader in deploy/host-health.mjs and had the launcher import it from there. The keeper test's release copy leaves out deploy/, so the keeper's launcher could not start (7 failures in keeper.test.mjs). The reader moved to telemetry/alerts.mjs, beside its writer, and host-health imports it.
    • The existing alert drill (telemetry-alerts.test.mjs) used a launcher fixture without the new series, so three new fleet rules went unknown in it. The fixture now carries them, quiet, as a launcher of this release does.
  • Long tail / externalities: per metrics scrape, one statfs and a histogram read in the launcher. Per healthz, one small file read (absent until the unit is enabled). About 2 ms more evaluation per collector scrape. No world is woken; no Soul or account data is touched. Labels are the world slug only.
  • Known limits:
    • V6 (the launcher sends anonymous viewers to the viewer thread) is not merged. sim_world_viewers reads the launcher's live map as it is on main; if V6 changes where viewers are counted, that map is the one place to keep in step.
    • worlds-created counts only rises the collector saw: creations before its first scrape are not counted.
    • Every world rule pages except refusals (see Deviations), so a world over tick-slow or gc-pressure makes hostOk false, as the design says. The release snapshot needs hostOk, so a release during a firing alert needs --accept-host-codes.
  • Docs: docs/operations/telemetry.md (series, rules, reaching a person, the unit, cost).

Deviations from the design

  • The disk series is two series. sim_host_disk_size_bytes sits beside sim_host_disk_free_bytes, because "under 20 % free" needs the total.
  • refusals does not page, like viewers-high. It fires at once on any single refused request, which is the ceilings doing their job. Paging on it would make the public health red on every rate limit.
  • The host problem code goes through <data>/alert-problems.json, written by the collector. The launcher's healthz reads it on every request, the way it reads the guard's world problems. deploy/host-health.mjs folds it every 10 min, which also logs ALERT lines. Healthz trusts its own read over the timer's copy, so a resolve clears at once. A dead collector (file older than 4 scrapes and 2 min) is alert:stale; a clean stop removes the file.
  • The unit finds the launcher from SIM_WORLDS_ARGS. collect --launcher-args reads the port and base from /etc/sim-worlds/<stage>.conf, so the unit carries no port.
  • Added by the coordinator, inside the purpose: a failed hourly backup (sim-worlds-backup-hourly@<stage>, from the hourly-backup PR not yet merged) raises backup-hourly:<stage> in host-health, like the nightly backup:<stage>. The health page alerts on it: a FAILED database from backup-hourly-last.json, or the unit's result for a timeout, which writes no record.

Outside Touches

  • tools/world-launcher/health-dash.mjs: 5 lines, for the hourly backup on the health page (the coordinator's addition).

Time

Started 07:57 UTC, PR at 08:17 UTC; ceiling 3 h.

Changelog

Labelled no-changelog: operator tooling (metrics, alerts, a unit, the health page); nothing changes for people using Souls.

Risk

Low until the unit is enabled: without alert-problems.json, healthz is unchanged. The launcher's additions are one histogram and one statfs per scrape. Once the unit is enabled, a firing alert makes the public hostOk false. That is the point, but a noisy rule would page; A rules.json override (--rules, added to the unit by a drop-in) can raise a threshold or turn a rule off without a release. To roll back, disable the unit (which removes the file) or revert the commit.

🤖 Generated with Claude Code

…and reach the public health check (#2541)

Four series (viewers per world, the launcher's own loop delay p99, worlds created, disk free/size), four rules
(launcher-loop-stall, worlds-created, disk-free, and viewers-high as information only), and a problems file the
collector rewrites after each scrape: every firing rule that pages is an alert:<rule> host code in /__sim/healthz
at once (hostOk false), cleared when it resolves; a dead collector is alert:stale. sim-worlds-telemetry@.service
is installed by install.sh and never enabled. A failed hourly backup (timeout or a FAILED database) raises
backup-hourly:<stage> and shows on the health page, like the nightly one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@genekogan genekogan added the no-changelog Nothing user-facing; no changes/ fragment needed label Nov 16, 2026
@genekogan
genekogan marked this pull request as ready for review November 16, 2026 08:17
@genekogan genekogan added the ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Proof check: proof-ok passes

Tier T1 from tools/world-launcher/deploy/host-health.mjs, tools/world-launcher/deploy/install.sh, tools/world-launcher/deploy/sim-worlds-telemetry@.service, tools/world-launcher/health-dash-backup.test.mjs, ... (node tools/proof/tier.mjs).
Required at T1 (docs/process/PROOF-POLICY.yml): focused node --test of touched files; typecheck of touched packages.

Check Status Detail
problem statement ok present
linked issue ok #2541
acceptance ok 3/3 met
carried from the issue ok all 3 issue criteria present
self-audit ok checks, failed attempts, long tail and limits filled
Acceptance criteria
Criterion Status Evidence
[x] A1 Tests for each series and each rule (fire and resolve) met node --test tools/world-launcher/telemetry-spike.test.mjs -> 7 pass, 0 fail. Covered: the four series and their absenc
[x] A2 Test: a firing rule makes /__sim/healthz report hostOk: false with met node --test tools/world-launcher/telemetry-spike.test.mjs "A2: a firing rule makes /__sim/healthz report hostOk false
[x] A3 docs/operations/telemetry.md documents the series, rules, the unit a met docs/operations/telemetry.md sections "The series a spike needs", "Rules" (four new rows and which rules page), "Reach

Proof links: 0 video, 0 image, no proof.json.

Updated on every edit, push, label and review. How to fill it: .claude/skills/journal/SKILL.md. T3-T4 also need the maintainer's sign-off (tools/proof/gate.mjs). A maintainer can waive with the proof:waived label.

@genekogan genekogan added merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
@genekogan
genekogan enabled auto-merge November 16, 2026 08:44
@genekogan
genekogan added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 16, 2026
@genekogan genekogan removed the merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) label Nov 16, 2026
@genekogan

Copy link
Copy Markdown
Member Author

Merge queue: this PR left the queue without merging (reason: MERGE_CONFLICT), so it left the train. The failing run is on the gh-readonly-queue commit in the Actions tab.

Fix it and add merge:ready again.

# Conflicts:
#	tools/world-launcher/launcher.mjs
@genekogan genekogan added merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
@genekogan
genekogan enabled auto-merge November 16, 2026 12:00
@genekogan
genekogan added this pull request to the merge queue Nov 16, 2026
@genekogan genekogan added ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
Merged via the queue into main with commit 2e434f6 Nov 16, 2026
53 of 59 checks passed
@genekogan
genekogan deleted the agent/2541-alerts-reach branch November 16, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) no-changelog Nothing user-facing; no changes/ fragment needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scaling H5 · The numbers a spike needs are exported, alerted on, and reach a person

1 participant